Skip to content

Dynamic IVF Index Implementation #223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ibhati wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Dynamic IVF Index Implementation #223

ibhati wants to merge 16 commits into from

Conversation

@ibhati
Copy link
Member

@ibhati ibhati commented Nov 18, 2025 •
edited
Loading

This PR enables support for Dynamic IVF (Inverted File) Index for SVS, enabling efficient vector search with dynamic insert and delete operations. The implementation includes comprehensive memory optimizations, thread configuration improvements, and full Python bindings.

Key Features

🚀 Dynamic IVF Index

  • Dynamic Operations: Support for adding and deleting vectors after index construction
  • Blocked Storage: Uses BlockedData with configurable block sizes for efficient memory management
  • Thread Control: Separate configuration for clustering threads, index threads, and intra-query threads

TODOs for IVF:

  • Enable efficient Load/Save (for both static and dynamic IVF)
  • Complete IVF documentation
  • Batch iterator with IVF
  • Benchmarking support for dynamic ivf

ibhati added 15 commits November 6, 2025 09:09
@ibhati
Enable IVF train only mode where centroid are built but data is not
9398042 
added in the cluster. Added comprhensive tests for IVF
@ibhati
Add cluster assignment functionality and tests
7505c68
@ibhati
Add integration for train only scenario
dbef61e
@ibhati
Merge remote-tracking branch 'origin/main' into ib/dynamic_ivf
44eb6d8
@ibhati
Merge remote-tracking branch 'origin/main' into ib/dynamic_ivf
05702a5
@ibhati
Minor fixes
6fa85c0
@ibhati
First attempt at dynamic index
149ff83
@ibhati
Merge remote-tracking branch 'origin/main' into ib/dynamic_ivf
3796c5d
@ibhati
Optimized search and add_points functions
8700c6d
@ibhati
Clang tidy
fcc8d44
@ibhati
Python bindings for dynamic IVF, first version
769e7ea
@ibhati
Added get_distance and consolidate
aef3044
@ibhati
Add get_distance support in static ivf index
8d24b99
@ibhati
Merge remote-tracking branch 'origin/main' into ib/dynamic_ivf
3bf3310
@ibhati
Entable intra_query_threads and separate clustering threads
60434e6
@ibhati ibhati changed the title Prepare IVF to support dynamic operations Dynamic IVF Index Implementation Dec 9, 2025
@ibhati ibhati marked this pull request as ready for review December 9, 2025 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ethanglaser ethanglaser Awaiting requested review from ethanglaser ethanglaser is a code owner

@ahuber21 ahuber21 Awaiting requested review from ahuber21 ahuber21 is a code owner

@mihaic mihaic Awaiting requested review from mihaic mihaic is a code owner

@yuejiaointel yuejiaointel Awaiting requested review from yuejiaointel yuejiaointel is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ibhati